Traditional multiple choice testing techniques to assess the extent of a person's knowledge in a subject matter include varying numbers of possible choices that are selectable by one-dimensional or right/wrong (RW) answers. A typical multiple choice test might include questions with three possible answers, where generally one of such answers can be eliminated by the test subject as incorrect as a matter of first impression. This gives rise to a significant probability that a guess on the remaining answers could result in a correct response. Under this situation, a successful guess would mask the true extent or the state of knowledge of the test subject, as to whether he or she is informed (i.e., confident with a correct response), misinformed (i.e., confident in the response, which response, however, is not correct) or being lacked of information (i.e., having no information). Accordingly, the traditional multiple choice one-dimensional testing technique is highly ineffectual as a means to measure the true extent of knowledge of the test subject. Despite this significant drawback, the traditional one-dimensional, multiple choice testing techniques are widely used by information-intensive and information-dependent organizations such as banking, insurance, utility companies, educational institutions and governmental agencies.
In response to the foregoing drawback, an information referenced testing technique was developed and pioneered by Dr. James Bruno of the University of California at Los Angeles. Information referenced testing (“IRT”) techniques extract the test subject's information response and confidence associated with it, undertake to reduce guessing, and effect a scoring profile that is resistant to the affects of guessing. IRT test queries are generally represented in a two-dimensional format to incorporate two key components of recognition and confidence as part of the test questions or queries in a multiple choice test. The test example below illustrates the general principle of the IRT technique.
1. How many legs does a cat have?
A. 3
B. 4
C. 5
2. What is the common logarithm of the number 3?
A. 43
B. 47
C. 0.047
3. How many states border the state of New Mexico?
A. 4
B. 5
C. 6
4. The Panama Canal connecting the Atlantic and Pacific Oceans generally runs
A. North-South
B. East-West
C. Northeast-Southwest
Instructions:
Each question has a predetermined point score of +30 if the answer is correct or −100 if the answer is incorrect.
If you can narrow your selection to two choices and eliminate the other choice as being an answer, mark your answer for the two choices: (A or B), (B or C), (A or C). Your response will be assigned a predetermined point score of +10 if the answer is correct or −100 if the answer is incorrect.
If you do not know the answer, you may choose not to respond, in which case, you will not receive any points.
When certain that an answer is correct, the test subject selects a response from one of the letters A, B, or C, which corresponds to the answer that the subject is confident to be correct. Thus the selection of a single letter indicates a high confidence level in the answer on the part of the test subject. If the response reflects the correct answer, a point score of 30 will be assigned. However, if the test subject selects one of the letters A, B, or C, which reflects a confidence in that response, a wrong answer for the response will return a score point of −100. This negative score point marks a state of misinformation and the subject is misinformed as to the substantive information pertaining to that query.
If the subject chooses not to select any of the letters provided, which indicates that he or she has no knowledge or information to provide a response, a zero score point will be returned.
Thus, with respect to the above sample queries, if the test subject answered the above questions I-B, 2-? 3-BC, and 4-B the test subject would be considered as being informed, uninformed, part informed, and misinformed respectively on these test queries.
As illustrated above, the point scoring system of the IRT for correct and incorrect responses for the confidence levels reflected in the subject's answers are (a) +30,−100 when confident or sure; (b) +10,−100 when between two choices, and (c) 0,0 when the subject is without any information. Under the IRT protocol, a maximum score would be achieved if and only if the test subject is “honest” and does not overvalue his or her information. Thus any incentives to guess at an answer are significantly discounted.
Currently, use of the IRT techniques are on a case-by-case, or batch processing, with test creation, administration, scoring and reporting, which use requires significant human interface, labor and logistic support. More significantly is that informational or education material databases are generally disassociated with the results of the test performance and results interpretation thus impeding remedial actions to reeducate or retrain. The affects of such case-by-case application are further amplified where the tests are to be conducted at various locations.
Accordingly, there is a need for a robust and easily managed integral knowledge assessment and learning system, which is deployable in a distributed computer environment for developing and administering knowledge assessment across chronological and geographical bounds. Such a networked testing system would eliminate batch IRT processing, provide for a wider distribution of test subjects in organizations, ensure full confidentiality of the employee, and allow a more detailed and intelligent learning system which is geared toward the true information need of the user.
Traditional multiple choice, one-dimensional (right/wrong), testing techniques are forced-choice tests. This format requires individuals to choose one answer, whether they know the correct answer or not. If there are three possible answers, random choice will result in a 33% chance of scoring a correct answer. One-dimensional scoring algorithms usually reward guessing. Typically, wrong answers are scored as zero points, so that there is no difference in scoring between not answering at all and taking an unsuccessful guess. Since guessing sometimes results in correct answers, it is always better to guess than not to guess. It is known that a small number of traditional testing methods provide a negative score for wrong answers, but usually the algorithm is designed such that eliminating at least one answer shifts the odds in favor of guessing. So for all practical purposes, guessing is still rewarded.
In addition, one-dimensional testing techniques encourage individuals to become skilled at eliminating possible wrong answers and making best-guess determinations at correct answers. If individuals can eliminate one possible answer as incorrect, the odds of picking a correct answer reach 50%. In the case where 70% is passing, individuals with good guessing skills are only 20% away from passing grades, even if they know almost nothing. Thus, the one-dimensional testing format and its scoring algorithm shift the purpose of individuals, their motivation, away from self-assessment and receiving accurate feedback, toward inflating test scores to pass a threshold.
Confidence-Based Assessments, on the other hand, are designed to eliminate guessing and accurately assess people's true state of knowledge. In the 1980s, Dr. James Bruno pioneered information referenced testing (IRT) in direct response to the foregoing situation. IRT is a two dimensional (recognition and confidence) test scoring procedure that places less emphasis on restrictive response environments (students can indicate “I don't know”). The formative evaluation is in two parts. The first part is to provide feedback for student learning. The second is to provide feedback to provide support for instructional programs. Based on a decision theory model of testing rather than a psychometric model, IRT was found to be especially valuable, acceptable and applicable for individual student assessment. A number of studies were conducted throughout the late 1980s and early 1990s, and research papers were published in peer reviewed journals detailing the results. The IRT procedure then employed an objective, optically scan-able, partial credit type of test scoring system that measured accurate information, misinformation, lack of information and partial information in a student knowledge base. IRT has also been used extensively by the FAA, Nuclear Regulatory Agency, and major utility companies in areas where misinformation could have serious legal, political and social consequences. In the past, applications of the IRT concept have relied on paper score sheets and computers with optical scan capabilities.
The IRT approach was implemented as a Confidence-Based Assessment (“CBA”) Testing System in the above-cited parent application Ser. No. 10/115,157, filed Apr. 3, 2002 which is incorporated into the present application by reference. This Confidence-Based Assessment approach is designed to eliminate guessing and accurately assess people's true state of knowledge. The CBA format covers three states of mind: confidence, doubt, and ignorance. Individuals are not forced to choose a specific answer, but rather they are free to choose one answer, two answers, or state that they do not know the answer. The CBA answer format more closely matches the states that test takers actually think and feel. Individuals quickly learn that guessing is penalized, and that it is better to admit doubts and ignorance than to feign confidence. Moreover, since CBA discourages guessing, test takers shift their focus from test-taking strategies and trying to inflate scores, toward honest, self-assessment of their actual knowledge and confidence. In fact, the more accurately and honestly individuals self-assess their own knowledge and feelings of confidence, the better their numerical scores.
Aspects of the present application refine the Confidence-Based Assessment approach by compiling a standard multiple choice test into a structured CBA. After individuals complete a CBA, their set of answers are used to generate a knowledge profile. The knowledge profile precisely segments answers into meaningful regions of knowledge, giving individuals and organizations rich feedback as to the areas and degrees of mistakes (misinformation), unknowns, doubts and mastery.